The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. The paperclip maximizer is the canonical thought experiment showing how an artificial general intelligence, even one designed competently and without malice, could ultimately destroy humanity. The thought experiment shows that AIs with apparently innocuous values could pose an existential threat. The goal of maximizing paperclips is chosen for illustrative purposes because it is very unlikely to be implemented, and has little apparent danger or emotional load (in contrast to, for example, curing cancer or winning wars). This produces a thought experiment which shows the contingency of human values: An extremely powerful optimizer (a highly intelligent agent) could seek goals that are completely alien to ours (orthogonality thesis), and as a side-effect destroy us by consuming resources essential to our survival.

Now I suppose it is possible that once an agent reaches a sufficient level of intellectual ability it derives some universal morality from the ether and there really is nothing to worry about, but I hope you agree that this is, at the very least, not a conservative assumption. For the purposes of this article I will take the orthogonality thesis as a given. So a smarter than human artificial intelligence can have any goal.

There are a lot of major problems today with tangible, real-world consequences. A short list might include terrorism, U.S.-Russian relations, climate change and biodiversity loss, income inequality, health care, childhood poverty, and the homegrown threat of authoritarian populism, most notably associated with the presumptive nominee for the Republican Party, Donald Trump. Yet if you've been paying attention to the news for the past several years, you've almost certainly seen articles from a wide range of news outlets about the looming danger of artificial general intelligence, or "AGI." For example, Stephen Hawking has repeatedly expressed that "the development of full artificial intelligence could spell the end of the human race," and Elon Musk -- of Tesla and SpaceX fame -- has described the creation of superintelligence as "summoning the demon." Furthermore, the Oxford philosopher and director of the Future of Humanity Institute, Nick Bostrom, published a New York Times best-selling book in 2014 called Superintelligence, in which he suggests that the "default outcome" of building a superintelligent machine will be "doom." Should we really be worried about a takeover by killer computers hell-bent on the total destruction of Homo sapiens?